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This article describes the NBS Vision 
System developed by the Sensory Intel- 
ligence Group of the Robotics System 
Division which is used in the Auto- 
mated Manufacturing Research Facility 
(AMRF). It discusses the objectives of 
the Vision System and its applications in 
the factory environment. Since the Vi- 
sion System is a multi-processor system, 
each process is described according to 
its position in the vision hierarchy as 
well as to its particular logical and com- 
putational functions. The interfaces be- 
tween the individual processes of the 
Vision System and the interfaces be- 
tween the Vision System and other 



AMRF systems are described. AMRF 
documentation packages describing the 
Horizontal Workstation, the Real-Time 
Control System and the Material Han- 
dling Workstation are available from the 
Center for Manufacturing Engineering. 
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Introduction 



The use of real-time sensory feedback in the 
manufacturing work place adds the dimension of 
intelligence to automated tasks. With the use of 
specially mounted cameras, visual information ex- 
tracted from a scene is made available to the appro- 
priate Automated Manufacturing Research Facility 
systems. This information enables robot control 
systems and planners to react appropriately to the 
tasks and objects in their environment. 

The NBS Vision System is an industrial parts 
recognition system which operates in real-time. It 
is a model based system which recognizes parts by 
matching an input image with a set of predefined 
models of parts. Models of a known set of indus- 
trial parts, computed in advance, reside in a data- 
base and are used to identify each instance of an 
object and to specify its position and orientation 



relative to a fixed camera configuration. This sys- 
tem is particularly suited for a factory application 
because the number of parts to be identified is 
small, the parts can be exactly specified with 
known tolerances on the dimensions, and the fea- 
tures of the parts, e.g., corners, holes, etc., are dis- 
tinctive. 

The NBS Vision System serves two sub-systems 
of the AMRF: it is used both in the Horizontal 
Workstation (HWS) and the Material Handling 
Workstation (MHWS). In the HWS, vision is used 
to confirm or reject the identity of a machine part 
and to compute the position and orientation of con- 
firmed parts. In the MHWS, vision is used to verify 
the contents of trays being delivered between 
workstations. 
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Physical Description of the NBS Vision 
System 

The NBS Vision System accepts images from a 
camera mounted on the wrist of a manipulator at 
the HWS as well as a stationary camera mounted 
on an inspection gantry in conjunction with the 
MHWS. Figure 1 describes the AMRF floor plan 
and the physical location of the NBS Vision Sys- 



tem in the factory. Both cameras are electronically 
attached to a hardware circuit that converts the 
incoming signals into black and white images and 
compresses that information as it is read. All visual 
information is extracted by analyzing this resultant 
binary image. 

The manipulator camera used at the HWS is 
fixed on the robot wrist. Below the camera, at a 
known distance from it, a light source is also at- 
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Figure 1. AMRF shop floor. 
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municates with other processes via a pre-defined 
common memory block. System routines have 
been developed to insure the integrity of this data 
transfer (fig. 4). 
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Figure 4. NBS vision system. 

Three of the processes operate in a bottom-up 
mode to read an image and globally extract infor- 
mation from it. These processes are First Stage Vi- 
sion (FSV), Second Stage Vision A (SSVA), and 
Second Stage Vision B (SSVB). A fourth 
process, the Multi-Level Database and Server 
(MLDSERV), acts in a top-down mode using its 
database of models to identify specific objects in a 
scene. The Supervisor Process (SUP) monitors Vi- 
sion System activity and receives commands from 
other AMRF systems via the Netboard (NET) pro- 
cessor. Commands for vision processing are passed 
from SUP to the Server which determines what a 
command is requesting and how to answer it. 

There are currently four types of requests to 
which the Vision System responds. In all cases, in- 
formation is supplied only when commands from 
either the RCS or the MHWS are received. The 
three questions posed by the RCS include a request 
for the position of the centroid of the object in 
view; a verification of a specific factory part in a 
scene, and if verified, the position and orientation 
of that part; and lastly, the computed range to the 
surface of the object in view. The answers supplied 
by the Vision System enable the RCS to compute 
the required robot motion to position the manipula- 
tor and plan a trajectory for picking up the verified 
object at the optimum grasp point. 



The fourth type of question to which the Vision 
System responds is posed by the MHWS and is a 
request for verification of the contents of a tray of 
parts being delivered between AMRF worksta- 
tions. Through the AMRF database, the MHWS 
supplies information regarding the intended con- 
tents of a particular tray, e.g., the number of parts 
contained in the tray and the sectors of the tray in 
which particular parts are expected to be located. 
By analyzing the image of the tray and comparing 
the actual tray's contents with the internal model 
database, this information is either confirmed or re- 
jected. A tray which fails verification is not pro- 
cessed further. 



Hierarchy of the Vision System 

First State Vision is the lowest level in the vision 
processing hierarchy and acts as the interface be- 
tween the camera hardware and the Vision System. 
It receives requests for raw image information 
from Second Stage Vision A and translates those 
requests into camera hardware commands. It then 
reads a compressed form of the camera image and 
transfers it to the appropriate common memory lo- 
cations. 

FSV controls the commands sent to the frame- 
buffer, which converts analog signals from the 
camera into digital intensity information. The re- 
sultant image is a rectangular array of image data 
or pixels which are assigned integer grey levels de- 
pending on their measured brightness. Threshold- 
ing converts these grey levels to two colors, black 
or white, depending on whether an individual pixel 
value is less than or greater than the given 
threshold value. The binary image is converted to 
run-length encoded information, a compressed data 
representation which records only transitions from 
black to white or white to black (fig. 5), and is read 
and stored by FSV. 
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Figure 5. Data representation, (a) Binary image, (b) Run-length 
representation. 
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tached to the wrist. The Hght source is able to gen- 
erate two kinds of light flashes. The first is a flood 
flash that illuminates the whole field of view of the 
camera while the second light source generates 
two parallel planes of light which illuminate those 
points in the field of view that intersect the planes 
of light (fig. 2) [1]. 

The flood flash is used to find shape features 
such as the outlines of objects in two dimensions. 
The structured light is used to compute the pitch 
and yaw orientations of simple geometric surfaces 
and the range to objects. A combination of struc- 
tured light and flood flash enables the system to 
extract elevation, azimuth, and roll orientation of 
the illuminated surfaces (fig. 3) [2], 

The camera mounted on the inspection gantry is 
provided with flood lights rather than a flash. 



Since this is a fixed configuration, the range to the 
objects in the scene is always known and thus only 
two dimensional shape features are required for 
analysis. 



Overview of the NBS Vision System 

The NBS Vision System is designed in a hierar- 
chical manner: commands from both the MHWS 
and the Realtime Control System (RCS) operating 
in the HWS are decomposed into lower-level tasks 
and executed by appropriate levels of the system. 
There are currently six independent processes op- 
erating asynchronously which analyze and extract 
information from an image scene. Each process 
resides on its own microprocessor board and com- 





(b) 



Figure 2. (a) Camera and light source mounted on the manipulator, (b) Relationship between the camera and the light sources. 






Figure 3. Left: Binary image of an object as seen under flood lighting. Center: Object as seen using the plane of light. Right: The two 
images superimposed. 
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SSVA receives its commands from the next 
higher level of the hierarchy, SSVB. It waits for 
FSV to read in an image and then reads the run- 
length encoded data from a predefined common 
memory area. SSVA performs different operations 
on the incoming data depending on the kind of pic- 
ture (flood or structured light) that has been re- 
quested. 

On receiving picture data from a floodlit image, 
SSVA performs a connected components analysis 
[7] and constructs a list of the objects (blobs) in the 
image. It also computes various properties of each 
blob such as the area, centroid, moments, etc. and 
encodes all points on the boundary. When the im- 
age results from using parallel planes of light, 
SSVA constructs groups of connected curve seg- 
ments [7]. The segments correspond to single 
curves or to pieces of curves starting where a 
curve splits and ending when it ends or merges 
with another curve. The results of these computa- 
tions are passed to SSVB by storing the structures 
into another common memory block. 

SSVB receives commands from the first level of 
the Multilevel Database Processor (MLDSERV) 
and passes them down to SSVA. Upon receiving 
the data structure of blobs or curves in the image, it 
performs feature analysis on each object in the im- 
age. For floodlit images, this involves finding the 
corners, principal axis, number of holes, and 
perimeter of each object. For structured light im- 
ages, each segment is described in terms of a poly- 
nomial and the endpoints. SSVB sends the list of 
objects and a set of structures describing the fea- 
tures to MLDSERV and then waits until another 
command is received. 

In addition to acting as the Vision System data- 
base, MLDSERV is also the module which per- 
forms recognition and matching tasks. As a 
database, it stores the results of the lower-level 
processing in structures appropriate for the upper- 
level recognition algorithms. It also contains a 
database of statistical information which has been 
computed a priori describing attributes of all parts 
to be accessed by the RCS and the MHWS. The 
Server is the portion of MLDSERV which inter- 
prets coinmands received by SUP and formulates 
an answer to the questions posed. Its main activities 
are concerned with recognizing objects and ex- 
tracting relevant information from the database. In 
addition, this module performs some of the range 
and position computations and is responsible for 
driving the graphic output displays. 

SUP is considered to be the "brains" of the Vi- 
sion System. It accepts commands from either the 



RCS or the MHWS, interprets them, translates 
them into a form suitable for the Vision System, 
and initiates the command sequence in the Vision 
System. If a poor quality image is detected, SUP 
tries to adjust the camera parameters to improve 
the picture and then repeats the process. If there 
are no processing errors, it sends the appropriate 
status and answers to either the RCS or the 
MHWS. In addition, SUP checks that each of the 
vision levels is working and monitors the terminal 
keyboard to service interactive features useful for 
providing information to the human operator. 



System Interfaces 

The Vision System processes interface with each 
other using a common memory area governed by a 
file system. The interprocess communications are 
handled using a library of system routines devel- 
oped at NBS [6]. The purpose of the library is to 
ensure data integrity of communications and to al- 
low for asynchronous communications between 
processes. Blocks of common memory, called 
"files", which are accessible to each vision process 
are defined. Flags associated with each file record 
the current state of each file. The state of a file 
includes information about the process which owns 
the file, the status of the file (open or closed), the 
last process which accessed the file, and the system 
time of last access. Only those processes granted 
read and/or write permission can access a file, and 
a process can only access a file if it is closed. The 
concept of opening and closing files is analogous to 
UNIX file system calls [4]. 

Communications between the RCS and the Vi- 
sion System are handled using a microprocessor 
communications board. A master-slave relationship 
is represented on the board to describe the relation- 
ship between the two systems and to indicate the 
direction of communication. Communication is ini- 
tialized by the RCS and protocols and command 
structures are used to allow the RCS to request 
service and to receive the necessary information 
from the Vision System. 

The MHWS and the Vision System communi- 
cate using the VAX common memory mailbox sys- 
tem [5], Mailbox areas are established when 
network communications are made to provide for 
message passing between systems. The Vision Sys- 
tem queries the mailbox command area for a 
change in sequence number to detect when a tray 
verification request has been sent. Upon comple- 
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tion of the recognition task, a status report is sent 
to the MHWS mailbox to close the transaction. 



Conclusion 

The NBS Vision System is currently being used 
in the AMRF to provide sensory information to 
the Realtime Control System and the Material 
Handling Workstation. Inputs are interpreted by a 
hierarchically organized group of microprocessors. 
The system uses knowledge of object prototypes to 
provide real-time feedback to the robot control 
system and to verify the contents of trays of ma- 
chine parts. Planes of light and flood light sources 
are used to quickly obtain the six degrees of free- 
dom of an object relative to the robot. The Vision 
System is then able to interpret an object's 2-D out- 
line in 3-space and to locate its centroid and princi- 
pal axis. This information is used to guide the 
robot's actions and trajectory as well as to identify 
specific parts. 
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